Notes on the RM projects 2024

Introduction: there was some confusion about the coins.  There are 211 different coins, split into 44 different types, since we can presume that coins were not transferred from person to person.  

EDA: Lots of space for low-information graphs (e.g., boxplots with just one box).  Some used a rule of thumb to detect outliers (not sure of the source of this, said to be Tukey), but anyway it can only be useful when the batch of numbers is of reasonable size (at least 20, say), since otherwise the IQR will be highly variable. 

`Outliers’ might be detected before the analysis, but it would make sense to perform analyses with and without them, to see if leaving them out actually makes any difference.  It doesn’t.

Figures/tables: Most captions were inadequate.   It should be possible to read the caption and understand the figure/table, without reference to the text.  In most cases this was impossible. Indeed, some reports had several identical captions (in which case the corresponding figures/tables should also be identical!).

Methods: No need to describe what a GLM is, or the basis of OLS or WLS. 

Explanatory variables: The obvious potential effects are Person, Coin and their interaction, plus Initial Side.  Some added Group (BSc, …) or Experiment, but this is essentially a partition of Person, so fitting Person + Group gives the same fit as Person (disappointingly, this seems not to have been obvious).  Others added some measure of time since the start of the experiment; there is no reason to expect this to be linear, so it might be fitted using splines, or low-order polynomials or inverse polynomials.  Others added some aspects of the physical coins (weight, size ..). 

In some cases linear models were incorrectly specified, for example by writing the linear predictor as $\beta_0 + \beta{\rm person}$, where presumably what is mean is $\beta_0 + \beta_{\rm person}$; the first would correspond to using Person as a numerical quantity, which would be silly, and the second to (correctly) treating Person as a factor.  Likewise with Coin, and the Person/Coin interaction.  

Care is needed with the interpretation of the estimates.  If the terms 1+Person are fitted in the default (corner-point) parametrisation $\beta_0 + \beta_{\rm person}$, then $\beta_0$ is the log-odds of success for the first person, and the log-odds of success for the others are of the form $\beta_0+\beta_i$.  Hence if fixed effects are used, then $\beta_0$ is NOT the average log-odds of success.  To compute the average success probability under such a model, one could compute $\hat\pi_i = y_i/m_i$ for person $i$, with corresponding variance estimate $v_i=\hat\pi_i(1-\hat\pi_i)/m_i$, then combine them using weighted least squares with weights $w_i=1/v_i$, giving overall estimate $ \hat\pi= \sum_i w_i\pi_i/\sum w_i$ with variance estimate $ v= 1/\sum w_i$ and large-sample normal-theory confidence intervals $\hat\pi \pm z_\alpha \sqrt{v}$.  Under a random effects model this is easier, because then $\beta_0$ represents the mean population log odds and can be taken at face value.   Similar comments apply to models with Initial Side, etc. 

Random/fixed effects:  The paper claims to be testing the argument from the DHM paper that there is a slight positive bias in coin tosses.  Since this applies to coin tosses as a whole, we probably should consider that the persons tossing the coins, and the coins themselves, might introduce slight further biases, and since both are drawn from populations, they should be treated as random effects.  This seems clear for the persons, but there are two ways to view the coins.  One might imagine that coins of different types would have slightly different biases, so that the effect $\beta_{\rm coin}$ would have 44 different levels.  These would presumably be fixed effects, since we would expect them to be the same for every coin of each type.  Or one might imagine that they correspond to different degrees of wear for every individual coin, and then there would be 211 random effects, since if we took further coins the effects would never be the same.  Or perhaps both $\ldots$.  

Fitting: The fitting of the binomial models was mostly correct.  Several reports applied WLS incorrectly, by setting the weight equal to the variance, rather than to its reciprocal.  In this case the estimates are correct (the primary assumptions of the analysis hold) but the standard errors, residual sums of squares, … were much too small (and this raised no red flags …).

One report wrote that ‘WLS cannot deal with nested or grouped data, as it cannot account for random effects or hierarchical structures’.  This confuses a fitting method (WLS) with a class of models, and moreover slide 240 explicitly allows for weights in random effects models to be fitted using weighted least squares, which is, of course, also ML estimation for normal errors. 

Model selection:  AIC, BIC, $R^2$ or other similar quantities are unsuitable.  AIC/BIC are for use when many models are initially on an equal footing and the goal of the analysis is prediction for a similar set of data, but here there are clear meanings and obvious interpretations to Coin, Person, etc..   A further issue is that AIC/BIC etc. for different types of models (binomial, WLS, mixed effects …) are on different scales, so they can’t be compared across model classes.  

Comparison of the models by analysis of deviance/variance is appropriate, not least because an adequate model will have deviance/residual sum of squares that can be compared with a chi-square variable.   The ANOVA shows strong person effects, smaller coin and person:coin effects, a nonlinear time effect and no effects of initial face or coin weight.  One or two tables for model choice (even those showing only AIC/BIC) did not include the degrees of freedom, which is vital information. 

Model checking: there is no reason to use raw residuals, especially in situations such as this where the variances of different observations are very different (because of different denominators for the binomial variables).  Some people seemed to think that observations with large Cook statistics should be dropped — that this is not the case was clearly stated in the lectures.  

Some people used tests from econometrics (e.g., the Breusch-Pagan test) to check for  heteroscedasticity; this is pointless, since the variances obviously vary greatly.

Learning effect:  generally the reports that discussed the learning effect did so fairly sensibly.  However the so-called learning effect may reflect selection.  It is striking that the largest success probabilities tend to be for small experiments and there is some asymmetry in the funnel plot.  This might suggest that small experiments that gave low success rates were thrown away, whereas the larger ones (with higher investment by the experimenters) were kept.  This might be another explanation for the `learning’ (how would this work?)

Conclusions:  Several people gave final analyses without standard errors for their estimated probabilities.  Some did not even give a final estimate of the probability.  Both cost marks.